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IMAGE PROCESSING UNIT AND IMAGE PROCESSING METHOD 

Qfioic&nours^r) OF the r i\JU£h)a:o/^ 

The present invention relates to an image processing unit and image procjessing 
method for computer graphics. 



Providing real and gorgeous images in a computer graphic (CG) system 
mapping is demanded. The easiest means for generating such real and gorgcoi]s 
using a large volume of texture data. 

However, in a system which executes high quality texture mapping such 
mapping, very high-speed access is demanded for the texture buffer, and to use 
volume of data, a large capacity high-speed storage device must be provided, 
enormously high cost, which makes it difficult to provide a large capacity storaj 
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With the forcg ei ng in view, it is an object of the present invention to pro v y ido - o n- 
4mage-proeessnig"imit-tmd4ntage"proeessing-met^^ images-using 
-a-fektively-small-capa©it^y-textiire-bttff^ 



X^Axi image processinfi-uiMt according to the present inveniioxi^Gos^j4s^ s 



device for storing texture data, a second storage device for storing a part of the 
data, and a processing section for executing image processing based on the 
above second storage device, where the above processing section updates the 
reading the texture data from the first storage device and writing that data to thej second 
storage device in a predetermined case. 
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Here Ihe predetermined case is a case when updating the texture data is accessary, 
such as the case when texture data not stored in ihe second storage device must used. 
The texture data may be updated in advance when processing capability is suffi< :icnL The 
storage devices arc not only semiconductor memories but include such external devices as 
HDD and CD-ROM. 

The image processin^writ according to the present invention is characte rized in that 
the above mentioned first storage device stores data including compressed textu -e data, and 
the above mentioned processing section ftirther eofflprises a data decomprcssior circuit fpr 
decompressing the read texture data so as to write the decompressed data to the second 
storage device. 

Here the texture data to be stored in the first storage device can be non-c impressed 
data, data containing both compressed and non-compressed data, or compressed data. 

The image processing wrft according to the present invention is characterized in that the 
above mentioned processing section furthcr^ mp rig os a first-in-first-out storage device for 
receiving the read texture data, temporarily storing this data, and outputting the 



lata to the 



above mentioned data decompression circuit. 

The image processin^^uftk according to the present invention is characte ized in that 
the above mentioned processing section fiirthcr G empriscs a pajette transformati >n circuit for 
executing palette transformation when the texture data is updated. 

The image processin^^mt-according to the present invention is characte; ized in that 
the above mentioned processing section fiirthci ^mprises a mip map gencratiorj circuit for 
generating a mip map when the texture data is updated. 

An image processing method according to the present invention uses a fiht storage 
device for storing texture data and a second storage device for storing a part of t le texture 



data, so as to execute image processing based on the texture data in the second i 
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and eefl^rises an updating step for updating the texture data by reading the texi|ure data from 



/3 



decomf resscd 



the first storage device and writing that data to the second storage device in a 
case, 

The image processing method according to the present invention further 
data decompression step for decompressing the read texture data when the data 
first storage device is compressed texture data, characterized in that the 
written to the second storage device. 

The image processing method according to the present invention further 
palette transformation step for executing palette transformation when the textun : 
updated. 

The image processing method according to the present invention further 
mip map generation step for generating a mip map when the texture data is u 
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Fig, 1 is a functional block diagram^f-i m - ^ tB ^ p r oc o s G iug unit in aceoiiJatice wi t h 
lh£Xinb€>dHBi^ 1 of the present invention; 



Fig. 2 is a functional block diagram of a geometry processor of the imag 



^^ ^4init in accordance wi t h the - embodiment 1 of the present invention; 

/)// 



Fig. 3 is a functional block diagram of a^geemctiy processor of the imag ; 
^^^jffliHft-accQfda nce with th e- embodim e nt - 1 of the present invention; 

Fig, 4 is a functional block diagram of a texture processor of the image 
jKP^e^J^ uait4iv«ceordanee-with-the-embQdinient-l- of the present invention: 

Pig- 5 is a functional block diagram of a shading processor of the image frocessing 



^^^nitjnjcf ^ rd an cft w i t h t he cmbo dHnenul-of the present invention; 
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Fig. 6 is a functional block diagram ^f the : image processing unit - in acco 4ance with 



A 



y%e-«mbodim"ent^ of the present invention; 

Fig. 7 is anymt£rn^-Me^s^agiaj mi - ef the image pr Qc cs siflg-imj l in acG or-< lancejadiiuhe^ 



embodimenW of the present invention; and 



ft i s e . xpl a na feei y diagram on ( i hr dat a rea d, d e r nmpressing a nd-wi tto-Uming^in^ 



p. 



now be 



.Equipment and method of ajii embodiment^ of the present invention wil 
explained. --^Hi^fembodiment % of the present invention has a mechanism to up( late texture in 
real-time at high-speed. 

Fig, 1 is a block diagra^^otaB4iBage^oce$$ing-unit4^ ihe 
^mbodiment-l of the present invention. In Fig. 1, 1 is a CPU (Central Process] ng Unit) 
which manipulates objects in a virtual space, receives information thereof, and executes 
various controls, 2 is a geometry processor, which executes the coordinate trai isfornialion 
of polygons, such geometric transformation (vector arithmetic operations) as cli ?ping and 
perspective transformation, and luminance calculation in three dimensional com puter 
graphics at high-speed, 2a is a polygon material light buffer RAM, which is a suffer for 
storing effective polygon data, maierial data and light data for one frame when X le geometry 
processor 2 executes processing. A polygon is a polygon constituting a three-c imensional 
body in a virtual space. A breakdown of data to be stored in the buffer memor f 2a follows. 

Link information, coordinate information and other attribute informatior of polygons. 

LINK X, LINK Y, X, Y, iz, Tx, Ty, Nx, Ny, Sign Nz, Alpha, Light ID, 1 /laterial ID 



etc. 
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Material information 

Depth enable, Depth function. Depth density. Texture enable. Fog enabl 
translucency enable, texture type, texture function, ofiEset x, y, size x, y, repeat > , y, mirror x, 
y, color id. Sine, Material specular, Material emission. Polygon color, Texture node, blend 
mode, etc. 

Light information 

Light Position, Light Direction, Light Type, Attenuation, Cutoff, Spole>4>. Light 
Color, Light Ambient, etc. 

3 is a fill processor for executing hidden surfece removal processing, 'the fill 
processor 3 fills a polygon in an area, and determines each information on the p )]ygon which 
is closest to the viewer for each pixel. 

4 is a texture processor. The texture processor 4 pastes texture on each! pixel in an 
area. Texture mapping is a processiag for creating an image by pasting (mapp ng) patterns 
(texture), which are defined separately from the shape, on the surface of an obje ;t for which 
shape has been defined. 4a is a texture RAM, where a texture map for ihc text jrc processor 
4 to execute processing is stored. 

5 is a shading processor. Shading is a method to express the shadow o( an object 
comprised of polygons while considering the normal line vector of a polygon, pbsition and 
color of a light source, position of view point, direction of line of sight, and other foctors. 

is a frame 
frame buffer 

5a, and after the digital data is converted to analog signak, the analog signals ar 3 supplied to 
such displays as a CRT, liquid crystal display and plasma display, which are noj depicted 
here. 



The shading processor 5 determines the luminance of each pixel in an area. 5* 
buffer for storing image dat^ o a muLflcg een. Data iis sequentially read from the 
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6 is a program work polygon buffer RAM fi3r storing the programs of th s fiPWi and 
commands to the graphic processor (e.g, database of polygons, display lists), fhis buffer 

CPU X: 

memory 6 is a work memory of the GPU4 as well. 

The fill processor 3, texture processor 4 and shading processor 5 executj rendering for 
creating pictures using models defined in the virtual space coordinates. In renhering, each 
area is processed sequentially from the upper left of the screen. Rendering pre cessing is 
repeated for the number of areas. 



Now detaDs on the image processin^^it in ^rrprdance with thc-embod 



present invention will be described with reference to the functiorjjblock diagran 
Fig. 5. 



ffleBt4-of the 



s in Fig, 2 to 
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Fig. 2 is a functional block diagram of the geometry processor 2. In Fij 21 is a 
data dispatcher, which reads a command from the buffer RAM 6, analyzes the c Dmmand, 
controls a vector engine 22 and clipping engine 24 based on the analysis result, 
the processed data to a sort engine 27, 

22 is the vector engine for executing vector arithmetic operations, Vec 
handled are stored in a vector register 23. 

23 is the vector register for storing vector data for the vector engine 22 ^ operate. 

24 is the clipping engine for executing clipping. 

25 is a Y-sorl INDEX, for storing the Y index for a sort engine 27 to exe|;:ute Y 

sorting, 

26 is an X-sort INDEX, for storing the X index for the sort engine 27 to Ixecute X 



sortmg. 



27 is the sort engine (sort engine) for searching a polygon to enter the tai get fragment 
in the buffer 6 by executing X sorting and Y sorting. The searched polygon is ; tored in the 
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buffer memory 2a, and is also sent lo the fill processor 3 for rendering. The scjrt engine 27 
also controls a polygon TAG 28 and a polygon cache 34. 

28 is the polygon TAG, which is a buffer for storing the TAG of the pel 

34, 

Fig, 3 is a functional block diagram of the ge on a o fay processor ^- In Fi^. 3, 31 is a 
cache controller for controlling the later mentioned material caches 42, AS, 51b, 52a and 53a, 
and a light cache 51a. 

32 is a material TAG for storing the TAG of the later mentioned materiajl caches 42, 
45, 51b, 52a and 53a, and light cache 51a. 

33 is a light TAG, which is a buffer for storing the TAG of the later raei^ioned light 
cache 51a, 

34 is a polygon cache, which is a cache memory of polygon data, 

35 is an initial parameter calculator, for determining the initial value of i)DA. 

36 is a Z comparator array for executing Z comparisons between polygo is for hidden 
surfece removal processing, and also for padding the polygon ID and internal r4ios tO, tl and 
t2. The Z comparator array 3^c ompricos 8x8=64 units of Z comparators. Si \cc these Z 
comparators operate in parallel, 64 pixels can be simultaneously processed. Ej icb Z 
comparator stores data on a polygon. For example, a polygon ID, iz, tO, tl, t2, window, 
stencil and shadow are stored, 

37 is a vertex parameter buffer, which is a buffer for storing parameters bf a vertex of 
a polygon. Corresponding to the Z comparator array 36, the vertex parameter puffer has a 
size capacity for 64 polygons. 

38 is an. interpolator for calculating the parameters of a pixel by interpol; Ltion based on 
the calculation result of the Z comparator array 36, tO, tl, t2, iz and the content \i the vertex 
parameter buffer 37. 
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Fig. 4 is a functional block diagram of the texture processor 4. In Fig. \, 41 is a 
density calculator for calculating the blending ratio fiDr fog or depth cueing. 

42 is a material cache for storing data on depth information. For exam )Ie, Depth 
enable, Depth function, Depth density. Depth end z. Texture enable, and Fog enable are 
stored. 

43 is a window register, which is a buffer for storing information on wir|dows. For 
example, k^, cz, fog function, fog density and fog end z are stored. 

44 is an address generator for calculating addresses on a texture map bas ed on the 
texture coordinates Tx;, Ty and LOD* 

45 is a material cache for storing data on material For example, transli icency enable, 
texture t^, offset x, y, size x, y, repeat x, y, minor x, y, and color id are storec . 



I 



yi ^^^ a TLMMI calculator (TLMMI: Tri Linear Nfap Mtip InlcrpOfa tion^br executing 
tri-linear mip map interpolation, which is three dimensional interpolation. Mif map is a 
technique for anti-aliasing when texture mapping is executed, that is, a techniquk for 
removing texture jaggies. Mip map is based on the following principle. Origin illy, the color 
(luminance) of an object face to be projected to one pixel must be a mean value 3f colors of 
the corresponding mapping area. Otherwise, jaggies become outstanding, whic h drops the 
quality of texture dramatically. However, if a mean value is determined for ea :h pixel, the 
calculation load becomes too high, which takes time to process, and which requ res a high 
speed processor- Mip map solves this problem. In mip map, a plurality of mi pping data 
having a width which is a multiple of 2 arc prepared in advance so as to simplify the 
tabulation of colors (luminance) of the mapping area corresponding to one pixel A size of 
the entire mapping area corresponding to one pixel is between any two data of these plurality 
of data having a width which is a multiple of 2. By comparing these two data, [he color of 
the corresponding mapping area is determined. For example, when screen A (: : 1) and 
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5, 51 is an 



screen B (x 1/2) exist, the pixels of screen A and the pixels of screen B correspo iding to each 
pixel of screen C (x 1/1.5) are determined respectively. At this time, the color \i the pixel 
of screen C is a color between the pixel in screen A and screen B, 

47 is a color converter for executing color conversion at 4 bit texcl. 
48 is a color palette where color information at 4 bit tcxel is stored. The coloi 
stores colors to be used for drawing graphics. Colors that can be used for one fixcl arc 
detenrdned corresponding to the content of the color^pallet 48. 

Fig- 5 is a fiinctional block diagram of the shading processor 5. In Fig. 
intensity processor for calculating the luminance ifor polygons after texture mapping. 

51a is a light cache for storing light information. For example. Light P )$ition. Light 
Direction, Light Type, Attenuation, Cutoff Spotexp. light Color and Light Arafient are 
stored. 

51b is a material cache for storing information on material For examp 
Material specular, materia] emission are stored 

51c is a window register for storing information on windows. For exanjiple. Screen 
center. Focus, Scene ambient are stored. 

52 is a modulate processor for executing associations of polygon color ajid texture 
color, intensity modulation and fog processing. 

52a is a material cache for storing information on materials. For examijle, Polygon 
color, and Texture mode are stored. 

52b is a window register, which is a buffer for storing information on windows. For 
example, Fog Color is stored. 

53 is a blend processor for blending data on the color buffer 54 and writiig the 
blended data to the color buffer 54. The blend processor 53 blends the current 
and the pixel color of the frame buffer based on the values of the blend rate regi 
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Dixel color 
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writes the blended data to ibe frame buffer of the bank indicated by the light bar k register. 
With the blend processor 53, residual image processing can be executed. 

53a is a materia] cache for storing information on materials. For cxamf le, blend 
mode is stored. 

54 is a color buffer, which is an 8x8 size color buffer. The color buffer 
double bank structure. 

55 is a plot processor for writing data on the color buffer 54 to the frame 

56 is a bitmap processor for executing bit map processing, 

57 is a display controller, which reads data of the frame buffer 5a, suppli 5S the data to 
DAC (Digital to Analog Converter), and displays the data on a display which is 
here. 



54 has a 



buffer 5a. 



not depicted 



Fig. 6 is a block diagraiCL of the equipm e nt shown in Fig. 1 - Fig, 5, creat sd for 
explanatory purposes. The image processing unit 101 is connected with the.Q yyi and 

exchanges commands and data with the-<^yi-, and accesses an external storage 

A ^ 



device 102, 



such as a hard disk, a buffer memory 103, a texture buffer 104 and a frame buff( r 5a, The 
texture buffer 104 stores texture maps. This texture buffer 104 executes high c uality texture 
mapping in real-time, therefore a high-speed operation is required. The buffer memory 103 
also stores compressed texture data along with other data. The buffer memory 103 is a 
general buffer memory, which has a slower speed and larger capacity then the t( xture buffer 
104. The image processing unit 4-ef4fee^mbodimenLl^of the present invcmioi i has a 
function for reading compressed texture data from the buffer memorj^^^xecuti ig data 
decompression, and writing that data to the texture buffer 104, in addition to ore inary 
processing. 



Fig- 7 is- a - b l o c k diagram of the image processing unit 101. — Fig. 7 corrt 



■fl ow - of -pre€e$j»mg. Numerals 112 - 119 are circuits for processing a conveniic nal three 



sponds to the 
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dimensional computer graphic system. These circuits are cascaded. Nuraera s 111, 120 - 
124, on the other hand, are circuits for executing thc^pFeeessing-of-th ^mbodim of the 
present invention. These circuits are cascaded, and the final output is input to he texture 
generation circuit 117 via the texture RAM interface 124. 

A coordinate transformation circuit 112 receives a command from a late mentioned 
command analysis circuit 111 and executes coordinate transformation for gener iting images. 
A clipping circuit 113 clips a portion outside the field of view after the data is tr msformed to 
an eye coordinate system. A perspective transformation circuit 114 transforms the eye 
coordinate system to the screen coordinate system. A fill circuit 115 fills trans formed 
images, A Z comparison circuit 116 compares Z values which indicate perspe :tive, and 
executes hidden surface removal by the Z sort method. A texture generation c rcuit 117 
generates texture based on the texture data. A color modulation circuit 1 18 am I a blend 
circuit 119 adjust the color of images. 

CPO X. 

A command analysis circuit 111 analyzes commands firom th^GPtft, di itinguishcs a 
command (data) for image generation and a command (data) for updating textuie data, and 
charmels the commands to the coordinate transformation circuit 112 and FIF^ rirsl-In-First- 
Out) 120. The FIFO 120 is a first-in-first-out memory for absorbing the differ mcc between 
the data read speed and the data decompression/write speed. HP^s a memoijyr used for 



data exchange between two parts which have different timing, and can adjust th 



even if timing is different at both ends of FIFO»^ A data decompression circuit 



difference 
121 is a 

circuit for decompressing the compressed data which was input, A palette trar sformation 
circuit 122 is a circuit for executing palette transformation, A MIP MAP gene alion circuit 
123 is a circuit for automatically generating a mip map. Mip map Is a techniqi e for anti- 
aliasing when texture mapping is executed, that is, a technique for removing jag gies. A 
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texture RAM interface (I/F) 124 is an interfece with a texture buflfer (RAM) 104, which has a 
mechanism for mediating between ordinary image processing and updating texti ire. 
The operation will now be explained. 

Data compression is very effective for such image data as a texture map. For 
example, a full color (16,000,000 colors) image can be expressed by 8 bit per te ce], since in 
most cases several hundred colors are actually used. In mip map, a plurality ol mapping 
data having a width which is a multiple of 2 is prepared to simplify the tabulatio i of colors 
(luminance) of the mapping areas corresponding to one pixel, and a mip map, w lich is a 
simple reduction of the original image, can be easily generated only if the origin al image is 
available. The image data often has the same color as the texel color of a ncarl ly area where 
many repeat patterns exist, therefore a general compression algorithm, such as r m length and 
slide dictionary, can easily be applied 



If the data is stored in a compressed state, however, decompressioii^^oc jssiag is 



h-speed, 



3, storing 
to update the 



required to use the data, which takes lime. Since texture data is accessed at hi^ 
compressed data is not easy to use, and the data must be stored in a non-compre ;sed state. 

With the foregoing in view, the embodiment 1 of the present invention n lakes high 
speed access possible and improves the efficiency of memory use by storing dat i in a non 
compressed state in the texture buffer 104, which must be accessed at high-spee 
compressed data in the other buffer (buffer memory 103), and has a mechanism 
texture buffer 104 in real-time. 

In Fig. 7, when a command (data) for image generation is transferred frcjm the 
command analysis circuit 111 to the coordinate transformation circuit 112, texti 
color modulation, and blending processing are executed based on coordinate tra isforraation, 
clipping, perspective transformation, filling, Z comparison, and non-compressec data in the 
texture buffer 104. 
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When a command (data) for updating the texture data is transferred from the 
command analysis circuit 111 to the FIFO 120, ihe following processing is exec ited. 

The FIFO 120 is a buffer to adjust the data read speed and data decompn jssion/write 

son 

speed. Data flow is different between reading and decompres^»g + writing. 



Since data 
1 byte at 1 



read involves simple memory access, the data transfer speed is constant, such as ] 
clock. But it is difficuh to estimate how many bytes the data will become whe \ the data is 
decompressed, because this depends on the compression method. For example . 1 byte data, 
which was input, may become 2 bytes or 0.5 bytes. As Fig. 8 shows, when 4 b /tes of input 
data A, B, C and D become 8 bytes, A'O, A'l, A*2, A'3, BC% D'O, D'l and D'2 , if the bus 
transfer capability of output is twice that of input, it seems that data volume will be balanced 
and no data wait time occurs. However, in actual processing, wait time occurs since data 
flow is not constant, therefore extra wait states must be created. In the^ exampl 5 in Fig. 8, 
for example, a wait state is required before reading the data B after reading the c ata A, and 



after reading the data D, respectively. Also two wait states arc required after / 
which is data after decompression. 



So the FIFO 120 adjusts the timing. In the HF^data written at any tiijie by the 



input side can be output at any time in the sequence of writing by the output sid|. Therefore 
the ^^Ul can issue a i 



L command regardless,the data decompression state in the ij lagc 
CPU I ^ ' 



processing unit 101, The^ CrUl does not have to distinguish a command (data \ for image 
generation from a command (data) for updating the texture data. 

When the data decompression circuit 121 receives a command (compres scd data) for 
updating the texture dat^^compresses the data, and returns the data to a non-c impressed 
state. The compressed data has been stored in the external storage device 102 br the buffer 
memory 103. 
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In a CG system using texture mapping, a large volume of texture data mtst 
generate real and gorgeous images most easily. Thcrcrorc there is an cnorraou 
types of compressed data pre-stored. On the other hand, there is much less tex 

which is actually used simultaneously. When the tejaure data in the texture 

CPU / 

insufficient, the CPU i issues a command to read the necessary compressed data 

buffer memory 103^ decompress the data, and writes the data to the texture 

CPUdL ^ 

Since thcjGPtH: knows which texture data to use and what kind of texture data 

CPV ± 

stored in the texture buffer 104, thc^ePlJt can easily issue the command by comparing 



be used to 
; number of 
ure data, 
buffer 104 is 
from the 
bufftr 104, 
as been 

the 



texture data to be used and the texture data stored in the texture buffer 104, 

Compression and decompres^^pcQ<i@^tf)g&are executed according to \ conmionly 
known algorithm. For example, run length encoding, sliding dictionary, Huffmajn, and 



discrete cosine transformation are used. For the format, JPEG combining 
transformation and Huffman, for example, is used. 

Run length encoding is an encoding method used when a repeat of the si 
appears frequently. The length of a pattern is called "the run". When the run 
data length can be shortened by encoding the run before transferring, rather thai 
the pattern as is. 

Sliding dictionary is a method where previous data is stored, and this 
data compression- This method is effective for mesh patterns. 

Huffrnan is a code to be created such that the average number of bits per 
becomes a minimum when quantized sample sets are encoded. If Huffman is 
transformation can be executed with great flexibility. For example, when colo 
such as in the case of images for animation, long patterns often appear while 
rarely appear. In this way, appropriate encoding is possible according to the 
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JPEG (Joint Photographic Experts Group) is a color still picture comprej sion system 
cslablishcd by a group promoting the st^indardizaiion of a color still picture con pression 
system. JPEG is widely used for multimedia applications which handle still pi ;:tures, such 
as on personal computers. 

For the texture data after data decompression processing is executed, pa] ette 
transformation is executed by the palette transformation circuit 122, Then the- tVHPWAP 



generation circuit 123 generates a mip map corresponding to the decompressed 



The texture data after data decompression p£@eessi&g is executed is writi en to the 



te3cture buffer 104 by the texture RAM interface 124. When the texture buffer 



lata. 



104 does not 



have an open area, the texture data overwrites other data. For example, data n< >t frequeney^ X 
used, old data, or data used a long time ago, can be selected to be overwritten^ 

In this way, according to,tfe^mbodiraen^of the present invention, virtually any 
texture data can be used, even if the capacity of the texture buffer is smaU. 

Since the texture data to be stored in the low speed buffer memory is coj npressed, 
memory space can be conserved. 

Also because the data decompression mechanism is included in the syst< m, non- 
compressed data flows only over the high-speed bus of the tejcture buffer 104, a id 
compressed data flows over the low speed bus, which allows sufficient use of th c transfer 
capability of the buses. 

In the above description of the present invention, data for arithmetic ope ations and 
processing of various memories, and the operation programs of the CPU are pre vided to the 
image processin^iMHt as a storage medium, such as a CD-ROM, cassette ROM, LD, FD and 
HD. The storage medium includes a medium used for communication. Such a< a server and 
communication lines of the Internet, LANs and personal computer communicai: 3ns, 
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The present invention comprises a first storage device for storing texture data and a 
second storage device for storing a part of the texture data, and has an updating step for 
reading texture data from the first storage device and writing that data to the sec ond storage 
device in a predetermined case, therefore many texture data can be used while using the small 
capacity second storage device. As a result, both low cost and high-speed pro< essing can be 
implemented. 

The present invention also has a first-in-first-out storage devfce, which rkceives the 
read texture data, temporarily stores this data, and outputs the data to the data d< compression 
circuit. Therefore a timing adjustment in data exchange is unnecessary. 
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